Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Windows platform in MoviePy #626

Closed
wants to merge 33 commits into from

Conversation

Julian-O
Copy link
Contributor

@Julian-O Julian-O commented Aug 8, 2017

Refactor of setup.py and continuous integration.

This is a significant change to the way Travis and setup.py work.

The primary driving force for these changes: MoviePy is buggy on Windows and very difficult to install on Windows. I think the root cause is that it is not tested on Windows, so OS-specific code is added without any feedback to the developer.

These changes not only help MoviePy run on Windows and install more easily on Windows (still messy!) but more importantly allow it to be automatically tested on Windows, even for changes submitted by Linux developers.

Changes

  • Includes several outstanding PRs.
  • Improve setup.py
    • Dependency Clean Up
      • Support a range of versions in setup.py - particularly taking advantage of Semantic Versioning, where available, to know which future versions will be compatible. This is especially useful when people are installing many packages or from binaries (which may have limited availability) - it prevents version clashes.
      • Remove unnecessary dependencies. For example, optional installs are not required for testing, so don't make it hard to test.
      • Add missing dependencies. Some dependencies were not recorded (e.g. youtube_dl)
      • Add Environment Markers. Not all packages support all platforms. Avoid trying to install them where it won't work.
    • Further integrate pytest into setup tests.
  • Improve Travis
    • Move heavy lifting to setup.py, following DRY principle
      • Let setup.py do as much installation as possible.
        • This separation isn't perfect. A few packages need to be manually upgraded because Setuptools isn't good at resolving it needs to upgrade an existing installation.
        • numpy remains problematic.
      • Let setup.py run the tests themselves.
    • Exercise the installer in tests. Test not only the code, but the installer.
  • Introduction of Appveyor.
    • Travis-CI doesn't support Windows, but Appveyor serves a similar role in the Windows space.
    • Unfortunately, the attempts to simplify Travis don't always work in the Windows world.
      • PIP won't install numpy, so conda is used.
      • Neither PIP nor conda will install ImageMagick (for ffmpeg)
  • Incidental fixes to platform-specific problems
    • references to Python 3 exceptions.
    • references to Linux's /tmp directory.
    • upgraded downloader to use requests package, because old versions of urllib had outdated certificates, and were rejecting one of the downloads.

Work to do

  • Register the main branch for Appveyor, so every pull-request and check-in is tested on Windows automatically.
  • Update the version number to a legal one, supported by setuptools. See Semantic Versioning for recommendations.
    • Then, change script to build and distribute Windows binaries to make installation simpler on Windows platforms.
  • Dockerfile and ez_setup should not have been affected by this change, but re-testing them would be a good idea.
    • Automatically testing them would be a great idea.
  • Add OSX support to the Travis script.
  • Submit Windows testing to coveralls data. (Windows percentage should be higher because several tests are skipped in Travis.)

* Python already has a feature for finding the temp dir. Changed
test helper to take advantage of it.

* Still outstanding: Several hard-coded references to /tmp appear in
the tests.

* Liberation-Mono is not commonly installed on Windows, and even when
it is, the font has a different name. Provide a fall-back for Windows
fonts. (Considered the use of a 3rd party tool to help select, but
seemed overkill.)
Building/finding binaries on Windows is non-trivial. Aallow some
flexibility in the path levels. (I don't want to force existing users
to upgrade, but new users should be allowed the later patches.)
Doesn't do anything yet. The work is done in the subclasses that need
it.

Also supports context manager, to allow close to be implicitly performed
without being forgotten even if an exception occurs during processes.
Demonstrate good practice in the examples.
…led.

The work should be done in close(). Deleting can be left for the garbage
collector.
Again, avoid depending on __del__. Add a context manager interface.
Use it lower down.
Was concerned that lambda might include a reference to reader that
wasn't cleaned up by close, so changed it over to an equivalent
self.reader. Probably has no effect, but feels safer.
Note: It does NOT close all the subclips, because they may be used
again (by the caller). It is the caller's job to clean them up.

But clips created by this instance are closed by this instance.
test_resourcereleasedemo exercises the path where close is not called
and demonstrates that there is a consistent problem on Windows. Even
after this fix, it remains a problem that if you don't call close,
moviepg will leak locked files and subprocesses. [Because the problem
remains until the process ends, this is included in
a separate test file.]

test_resourcerelease demonstrates that when close() is called, the
problem goes away.
* Without tests changes, many of these existing tests do not pass on
Windows.
Helvetica wasn't recognised by ImageMagick. Changing to another
arbitrary font that should be available on all Windows machines.
Also changed test to meet Issue Zulko#598, but that is also being done in
PR#585, so will require a merge.
…sues."

This reverts commit dc4a16a.

I bundled too much into one commit. Reverting and reapplying as two separate commits for better history.
Removed as it was crashing on Windows, achieving nothing on Linux.
* Add key support for close()

   * FFMPEG_VideoWriter and FFMPEG_AudioWriter: Support close() and context managers.
   * Clip: support close() and context manager. Doesn't do anything itself. The work is done in the subclasses that need it.
   * Clip subclasses: Overrride close.
       * Move away from depending on clients calling__del__(). Deleting can be left to Garbage Collector.
   * CompositeVideoClip: Note: Don't close anything that wasn't constructed here. The client needs to be able to control the component clips.
   * AudioFileClip:  Was concerned that lambda might include a reference to reader that wasn't cleaned up by close, so changed it over to an equivalent self.reader. Probably has no effect, but feels safer.

* Update tests to use close().

   * Note: While many tests pass on Linux either way, a large proportion of the existing unit tests fail on Windows without these changes.
   * Include changes to many doctest examples - Demonstrate good practice in the examples.
   * Also, migrate tests to use TEMPDIR where they were not using it.
   * test_duration(): also corrected a bug in the test (described in Zulko#598). This bug is also been addressed in Zulko#585, so a merge will be required.

* Add two new test files:

   *  test_resourcereleasedemo exercises the path where close is not called and demonstrates that there is a consistent problem on Windows. Even after this fix, it remains a problem that if you don't call close, moviepg will leak locked files and subprocesses. Because the problem remains until the process ends, this is included in a separate test file.]
   * test_resourcerelease demonstrates that when close() is called, the problem goes away.

* Update documentation to include usage tips for close()

Not included:

    *  Example code has not been updated to use close().
Also, make runnable directly (to help debugging)
Old versions of urlretrieve have old certificates which means one of the
video downloads was failing.

Also requires changes to setup.py, to come.
Including adding ranges, removing unnecessary entries, adding missing
entries, adding environment markers, changing versions, and updating
pytest parameter handling.
Julian-O and others added 3 commits August 9, 2017 01:33
Remove conditional installations repeating the rules in setup.py
Remove some installation of test needs repeating the rules in setup.py
Add testing of installation options.
@coveralls
Copy link

coveralls commented Aug 8, 2017

Coverage Status

Coverage increased (+0.5%) to 54.727% when pulling be25f30 on Julian-O:WindowsSupport into 5a3cb6e on Zulko:master.

@Julian-O Julian-O mentioned this pull request Aug 8, 2017
optional_reqs = [
"opencv-python>=3.0,<4.0; python_version!='2.7'",
"scikit-image>=0.13.0,<1.0; python_version>='3.4'",
"scikit-learn; python_version>='3.4'",

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't looked at all of the changes here. So could be missing something. Is this suppose to be a test requirement still? Did it become an optional requirement as well?

Copy link
Contributor Author

@Julian-O Julian-O Aug 8, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I've understood what you are referring to, but I'll explain what I have done, and hope I answer your question through a scattergun!

MoviePy has a number of optional requirements - if we look at a feature like resize(), it will look for any of several different packages to provide the functionality it needs.

Previously all the optional requirements were considered part of the test requirements. If you asked setup.py to execute the tests, they were all installed. However, this was (a) unnecessary, and (b) painful to get to work on all platforms.

Now, the test_reqs doesn't contain as many requirements - it should only include requirements necessary to run the tests successfully. The tests are all succeeding.

The .travis.yml file does another test: that all the optional requirements can be installed via pip, so we know the setup.py doesn't name imaginary packages. I don't attempt that on the Windows version, as setuptools is incapable of installing all of the optional packages, such as scikit-learn.

This was referenced Aug 8, 2017
@tburrows13 tburrows13 added the breaking-change Must not merge without proper approval. Requires full documentation (own section) in the changelog. label Aug 24, 2017
@tburrows13
Copy link
Collaborator

These were all merged in with #630.

@tburrows13 tburrows13 closed this Feb 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking-change Must not merge without proper approval. Requires full documentation (own section) in the changelog.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants